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This is in response to the appeal brief filed 9/24/2005 appealing from the Office action mailed 
4/26/2005. 
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(1) Real Party in Interest 

A statement identifying by name the real party in interest is contained in the brief. 

(2) Related Appeals and Interferences 

The examiner is not aware of any related appeals, interferences, or judicial proceedings 
which will directly affect or be directly affected by or have a bearing on the Board's decision in 
the pending appeal. 

(3) Status of Claims 

The statement of the status of claims contained in the brief is correct. 

(4) Status of Amendments After Final 

The appellant's statement of the status of amendments after final rejection contained in 
the brief is correct. 

(5) Summary of Claimed Subject Matter 

The summary of claimed subject matter contained in the brief is correct. 

(6) Grounds of Rejection to be Reviewed on Appeal 

The appellant's statement of the grounds of rejection to be reviewed on appeal is correct. 
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(7) Claims Appendix 

The copy of the appealed claims contained in the Appendix to the brief is correct. 

(8) Evidence Relied Upon 

6,243,713 Bl NELSON et al. 6-2001 

6,748,382 Bl MOHAN et al. 6-2004 

(9) Grounds of Rejection 

The following ground(s) of rejection are applicable to the appealed claims: 

Claim Rejections - 35 USC §103 
The following is a quotation of 35 U.S.C 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 1-2, 4-8, and 10-17 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Nelson et al. (hereinafter "Nelson"), US 6,243,713 Bl filed 8/24/1998. 

Regarding independent claim 1, Nelson teaches analyzing a multimedia document to 
identify at least one markup tag containing a reference to a given one of resources in fig. 1-2, fig. 
4, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 9 line 7. Nelson shows in the col. 
5 lines 39-51 that the multimedia document can be an HTML document and HTML documents 
generally represent Web pages, thus the scope of Nelson includes multimedia Web page 



Application/Control Number: 09/496,086 Page 4 

Art Unit: 2176 

documents. Nelson also specifically indicates in col. 5 lines 31-38 and col. 8 lines 26-27 that the 
multimedia document may be a downloaded Web page. Nelson teaches selecting and executing 
a media processing program for analyzing the content of the multimedia data supplied by the 
given one of the resources to generate metadata describing the content in fig. 2, fig. 4-7, and col. 
9 line 9 - col. 14 line 55. The cited text section described how Nelson generates metadata for 
each of the types of multimedia content components. Nelson teaches formatting the metadata 
into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 

Nelson does not teach combining the Web page and the generated annotation (Nelson 
calls it a token) to form an enhanced Web page. As stated earlier, Nelson does teach that all of 
the tokens extracted from a particular document are linked to that document through use of a 
unique document ID in col. 15 lines 24-36. Therefore, tokens of text, image, video, audio, and 
other data are all linked together to represent a single document through use of the unique 
document ID. Thus, this is essentially the same in function, but only similar in form to the 
claimed limitation. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have modified Nelson to have created the claimed invention. It would 
have been obvious and desirable to have modified Nelson to have inserted the token 
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representations of image, video, audio, and other multimedia components into the text of the 
multimedia document so that the text did not have to be processed, thus allowing the extraction 
software design to be simpler. 

Regarding dependent claim 2, Nelson teaches determining the particular data type of 
the multimedia data supplied by the given resource and selecting a processing program for 
analyzing multimedia data formatted in accordance with the particular data type in fig. 2, fig. 4, 
and col. 3 lines 21-27. 

Regarding dependent claim 4, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson teaches that the position data may be used in creating the tokens, thus 
Nelson teaches that the additional metadata may be supplied by a source other than the content of 
the multimedia data. 

Regarding dependent claim 5, Nelson teaches wherein at least some of the additional 
metadata includes information obtained from the one markup tag in fig. 2, fig. 4-7, and col. 9 line 
9 - col. 14 line 55. Nelson indicates in col. 9 line 9 - col. 10 line 37 that the tags may be used to 
identify a position and a name for the identified multimedia component. 

Regarding dependent claim 6, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson does not specifically teach wherein the given resource is accessed 
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through the operating system of the computer which provides the given resource and wherein at 
least some of the additional metadata includes information obtained from the operating system. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have modified Nelson to have created the claimed invention. Nelson teaches the 
multimedia content components may be referenced from the multimedia document and thus a 
valid reference would be a multimedia content component from the operating system. It would 
have be obvious and desirable to have obtained a referenced resource from the operating system 
and used metadata from the operating system to describe the resource so that multimedia 
components could have taken advantage of the local computer's operating system resources. 

Regarding dependent claim 7, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson teaches in col. 5 lines 39-51 that the multimedia component may be 
referenced by tags in the multimedia document. If the reference indicates a resource located on 
the Internet, Nelson is then going to obtain additional information about the resource via the 
Internet in forming the characterization token. 

Regarding independent claim 8, Nelson teaches a parser for identifying markup tags in 
a Web page in fig. 1-2, fig. 4, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 9 line 
7. Nelson shows in the col. 5 lines 39-51 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
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teaches analyzing the content of multimedia data identified by markup tags to generate metadata 
describing the multimedia data in fig. 2, fig. 4-7, and col. 9 line 9 - col. 14 line 55. The cited 
text section described how Nelson generates metadata for each of the types of multimedia 
content components. 

Nelson teaches translating the metadata into a character-based text annotation describing 
the multimedia data in fig. 6-8 and col. 3 lines 19-59. Nelson teaches formatting the metadata 
into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 

Nelson does not teach storing the combination of a copy of the Web page and the 
annotation to form an enhanced Web page suitable for processing by text-based indexing and 
searching facilities. As stated earlier, Nelson does teach that all of the tokens extracted from a 
particular document are linked to that document through use of a unique document ED in col. 15 
lines 24-36. Therefore, tokens of text, image, video, audio, and other data are all linked together 
to represent a single document through use of the unique document ID. Thus, this is essentially 
the same in function, but only similar in form to the claimed limitation. It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to have modified 
Nelson to have created the claimed invention. It would have been obvious and desirable to have 
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modified Nelson to have inserted the token representations of image, video, audio, and other 
multimedia components into the text of the multimedia document so that the text did not have to 
be processed, thus allowing the extraction software design to be simpler. 

Regarding independent claim 10, Nelson teaches identifying one or more markup tags 
in a Web page which respectively identify one or more external resources which provide 
multimedia data in fig. 1-2, fig. 4, col. 2 lines 19-37, col. 5 lines 1 1-51, and col. 8 line 1 - col. 9 
line 7. Nelson shows in the col 5 lines 39-5 1 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches generating metadata which describes the multimedia data in fig. 2, fig. 4-7, and col. 9 
line 9 - col. 14 line 55. The cited text section described how Nelson generates metadata for each 
of the types of multimedia content components. 

Nelson teaches translating the metadata into a character-based text annotation describing 
the multimedia data in fig. 6-8 and col. 3 lines 19-59. Nelson teaches formatting the metadata 
into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 
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Nelson does not teach inserting the annotation into the Web page to form an enhanced 
Web page suitable for processing by a character-based text processing system. As stated earlier, 
Nelson does teach that all of the tokens extracted from a particular document are linked to that 
document through use of a unique document ID in col. 15 lines 24-36. Therefore, tokens of text, 
image, video, audio, and other data are all linked together to represent a single document through 
use of the unique document ID. Thus, this is essentially the same in function, but only similar in 
form to the claimed limitation. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to have modified Nelson to have created the claimed invention. 
It would have been obvious and desirable to have modified Nelson to have inserted the token 
representations of image, video, audio, and other multimedia components into the text of the 
multimedia document so that the text did not have to be processed, thus allowing the extraction 
software design to be simpler. 

Regarding dependent claim 11, Nelson teaches first identifying markup tags in a Web 
page and extracting the uniform resource locator (URL) of one of the external resources from the 
at least selected ones of the markup tags in fig. 1-2, fig. 4-7, col. 2 lines 19-37, col. 5 lines 11-51, 
and col. 8 line 1 - col. 14 line 55. Nelson shows in the col. 5 lines 39-51 that the multimedia 
document can be an HTML document and HTML documents generally represent Web pages, 
thus the scope of Nelson includes multimedia Web page documents. Nelson also specifically 
indicates in col. 5 lines 31-38 and col. 8 lines 26-27 that the multimedia document may be a 
downloaded Web page. Nelson teaches in col. 5 lines 39-51 that the multimedia component may 
be referenced by tags in the multimedia document. The reference to a multimedia component in 
an HTML Web page is a URL. 
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Regarding dependent claim 12, Nelson teaches retrieving multimedia data from one or 
more external resources and analyzing the content of the multimedia data to extract the metadata 
therefrom in fig. 1-2, fig. 4-7, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 14 line 
55. Nelson shows in the col. 5 lines 39-51 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches in col. 5 lines 39-51 that the multimedia component may be referenced by tags in the 
multimedia document. The reference is to an external resource from which the multimedia data 
is retrieved and the metadata extract therefrom. 

Regarding dependent claim 13, Nelson teaches identifying the data type of the 
multimedia data from each of the resources and then selecting a processing routine for 
multimedia of the identified data type from each of the resources in fig. 2, fig. 4, and col. 3 lines 
21-27. 

Regarding dependent claim 14, Nelson teaches indexing the multimedia document into 
a unified multimedia index to equally represent in a normalized fashion all of the tokens of the 
multimedia document and provide access to the Web page in response to queries expressing one 
or more attributes expressed in the text annotation in fig. 8-9 and col. 14 line 56 - col. 15 line 58. 
Each of the tokens is linked to the document from which it was extracted through use of a unique 
document ID referencing that document. 

Regarding dependent claim 15, Nelson teaches searching the content of the Web page 
in response to a search request to determine if attributes expressed in the search request are 



Application/Control Number: 09/496,086 Page 1 1 

Art Unit: 2176 

contained in the text annotation in fig. 11-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - 
col. 27 line 3. 

Regarding dependent claim 16, Nelson teaches indexing the Web page to provide 
access to the Web page in response to queries expressing one or more attributes expressed in the 
text annotation in fig. 1 1-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - col. 27 line 3. 

Regarding dependent claim 17, Nelson teaches searching the content of the Web page 
in response to a search request to determine if attributes expressed in the search request are 
contained in the text annotation in fig. 1 1-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - 
col. 27 line 3. 

Claims 3 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Nelson et al. (hereinafter "Nelson"), US 6,243,713 Bl filed 8/24/1998 in view of Mohan et al. 
(hereinafter "Mohan"), US 6,748,382 Bl filed 12/1/1999 with priority to provisional filed 
1/28/1999. 

Regarding dependent claim 3, Nelson does not teach generating a text data annotation 
expressed in accordance with the Extensible Markup Language. Mohan does teach generating a 
text data annotation expressed in accordance with the Extensible Markup Language in fig. 3 and 
col. 2 lines 16-28. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Mohan into Nelson to have created the claimed invention. 
It would have been obvious and desirable to have enhanced the tokens of Nelson with the XML 
based media asset management descriptions of Mohan to have ensured maximum 
interoperability as described by Mohan in col. 2 lines 27-28. 
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Regarding dependent claim 9, Nelson does not teach generating a text data annotation 
expressed in accordance with the Extensible Markup Language. Mohan does teach generating a 
text data annotation expressed in accordance with the Extensible Markup Language in fig. 3 and 
col. 2 lines 16-28. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Mohan into Nelson to have created the claimed invention. 
It would have been obvious and desirable to have enhanced the tokens of Nelson with the XML 
based media asset management descriptions of Mohan to have ensured maximum 
interoperability as described by Mohan in col. 2 lines 27-28. 

(10) Response to Argument 

Regarding Appellant's arguments in pages 6-9 that Nelson does not teach or suggest all 
of the limitations of independent claims 1, 8, and 10, the Examiner respectfully disagrees. 
Nelson teaches in col. 1 line 32 - col. 2 line 1 1 that both text and image retrieval databases are 
known, but generally operate independently of each other. Nelson teaches a motivating desire in 
col 2 lines 21-16 to provide a system, method, and software product that retrieves compound in 
response to queries that include various multimedia elements in a structured form, including text, 
image features, audio, or video. Nelson provides a solution to this need with its invention by 
creating character-based annotations called tokens. All the tokens created from the various 
dissimilar multimedia types are normalized into the same character-based format, thus allowing 
unified indexing and searching of the documents and their various types of multimedia 
components. The tokens are permanently associated or attached to the document from which 
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they were extracted via use of a document ED which is part of the token as is taught in col. 15 
lines 24-36. Thus, the token annotates the document from which it was created. 

Nelson describes that the documents and their components are indexed into a unified, 
common index in col. 2 lines 19-27. Since the tokens are character-based annotations, the 
unified index is created with a conventional text-based indexing system. Thus, the Examiner 
believes this indexing system used by Nelson is the same as that of the claimed invention once 
the system has obtained the character-based text annotations. The only prior difference is 
whether the indexing system obtains the annotations via their association with the document as is 
specifically taught in Nelson, or whether the indexing system pulls the annotations directly out of 
the document as is done in the claimed invention. Nelson calls its indexing system a multimedia 
index because of the logic of the character-based tokens representing multimedia objects. The 
fact remains that the tokens themselves are character-based annotations of the document they are 
associated with via their document ID. Thus, Examiner maintains that token annotations of 
Nelson teach the character-based text annotation of the claimed invention. Nelson does not 
specifically anticipate the claimed invention because Nelson maintains the token separate from 
the document and permanently attaches it to the document via the document ID attribute of the 
token. The Examiner believes, however, that the teachings in Nelson suggest to one of ordinary 
skill in the art at the time of the invention a desirability to imbed the annotation, or token, 
directly into the document. 

Nelson teaches in col. 5 lines 39-51 that it is used for indexing multimedia compound 
documents such as HTML documents, which are the standard document format for displaying 
web page documents. The HTML standard itself teaches the meta element, which is a character- 
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based text annotation inserted into the document. Conventional search indexes can use these 
meta element character-based text annotations to classify the document. Therefore, Nelson has 
knowledge of the advantages of imbedding character-based text annotations directly into the 
document to simplify the indexing of the document. Therefore, the Examiner maintains the 
belief that Nelson suggests, via knowledge of HTML documents, to one of ordinary skill in the 
art at the time of the invention that it was advantageous to have imbedded character-based text 
annotations directly into the document itself. This would have motivated one of ordinary skill in 
the art at the time of the invention to have considered modifying Nelson by imbedding rather 
than linking the character-based tokens into the multimedia documents, thus containing the 
metadata annotations within the document as is taught by the meta element of HTML. By 
embedding all of the tokens into the document, all of the meta data would have been extractable 
form a single source, the parent document, thus simplifying the software extraction of the 
character-based text annotations for indexing the multimedia document. Such a modification to 
Nelson would not render it unsatisfactory for its intended purpose because the intended purpose 
of Nelson is to provide a unified index of the multimedia document and all of its multimedia 
components and this is performed by the creation and indexing of the character-based tokens 
describing the content of the multimedia document and each of its multimedia components. 
Nelson fulfills its purpose of providing a unified index regardless of whether the character-based 
tokens are associatively linked to the document, or imbedded within the document. The indexing 
of the tokens would be the same regardless of where they are stored and Nelson would 
successfully commonly index the multimedia documents and their multimedia components either 
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way. It is for these reasons that the Examiner maintains that Nelson teaches or suggests every 
limitation of the invention as presented in claims 1, 8, and 10. 

Regarding Appellant's arguments in page 10 that Nelson does not teach or suggest every 
limitation of dependent claim 6, the Examiner respectfully disagrees. Nelson teaches in col. 1 1 
lines 41-53, col. 12 line 66 - col. 13 line 7, and col. 13 lines 62-67 that the format is determined 
for each media component and is then converted into a standard format and size. Therefore, 
Nelson teaches that the operating system is used by the indexing system to properly identify the 
multimedia components during the creation of the tokens corresponding to those multimedia 
components. Nelson explains that the format and size of the multimedia components are 
standardized, thus the operating system information is not specifically used as metadata. 
However, since Nelson teaches that the operating system information is used to produce the 
metadata, the Examiner believes Nelson recognizes the benefit of collecting information from 
the operating system. Nelson therefore suggests that the this metadata would be useful as actual 
information in character-based tokens if the multimedia data was not otherwise standardized. 
Thus, the Examiner maintains that Nelson suggests to one of ordinary skill in the art at the time 
of the invention the limitation of dependent claim 6. 
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(11) Related Proceeding(s) Appendix 

No decision rendered by a court or the Board is identified by the examiner in the Related 
Appeals and Interferences section of this examiner's answer. 

For the above reasons, it is believed that the rejections should be sustained. 
Respectfully submitted, 






William Bashore 





